


Practical Neural 
Networks (4) 


Part 4 (final): applications and large neural nets 


By Chris MacLeod and Grant Maxwell 





In this final part of the series, we will consider some of the other 
applications of neural nets and have a look at a few of the more difficult 
questions facing neural net researchers. 


During the series we've concentrated mostly 
on image recognition. This is because it is not 
only one of the most important applications 
of neural nets, but also the one that’s easy to 
illustrate, as we can “see” the images. 

However, as explained in Part 1, the 
neural net can be considered as a universal 
logic system, capable (providing that the net- 
work has three layers) of learning to produce 
any truth table you like, see Figure 1. 

If the output of the neurons is a sigmoid 
function, then it acts rather like ‘fuzzy logic’, 
and is able to produce analogue outputs — 
which can be useful for handling problems, in 
the real world, which are not ‘black and white’. 


Applications in Robotics 


To illustrate how a trainable truth table is 
useful, consider a robot controller as shown 
in Figure 2. 

The network can learn to make the robot 
behave as we wish if we train it to respond 
to the sensor inputs and produce outputs to 
control its motors. This is a simple example of 
what is termed an Artificial Nervous System 
(ANS) and an animal-like robot controlled 
through a learning control system is some- 
times referred to as an Animat. 

You could teach this robot to behave as 
you wish with the Back Propagation Algo- 
rithm. Alternatively, you could have it learn 
to survive by itself, using a re-enforcement 
learning algorithm. Re-enforcement methods 
work by rewarding behaviour which benefits 
the robot (e.g., avoiding obstacles) and 
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Figure |. Universal trainable logic. 
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Figure 2. Robot controller. 


strengthening the weights which 
caused this behaviour. This can be 
accomplished automatically in the 
robot itself (of course we need to 
make some judgements about 
what's good and bad behaviour and 
program these in, so that the robot 
has a value system). 

A slight variant on this is to use a 
Genetic Algorithm to evolve network 
types from a population which lead 
to good behaviour — this algorithm 


Drive to right wheel 
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is called Evolutionary Algorithm for 
Re-enforcement Learning (EARL). 


Applications in Control 


We can extend the control aspects 
mentioned above to other systems. 
For example, suppose we want a 
system to control a DC motor, see 
Figure 3. 

You might recognise the inputs 
as the typical PID types used in 
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Figure 3. Motor control. 
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Figure 4. A sampled waveform fed to a neural net. 


conventional controllers. A neuro- 
controller like this might learn with 


Man Machine Interface 
and Intelligent Sensing 


BP from a conventional controller, 


by using its outputs as targets. 


Alternatively, and more interest- 
ingly, it could learn from experience, 
using a Re-enforcement Learning 
Algorithm or a Genetic Algorithm to 


set its weights. 





In the control system above, the net- 
work responded to sensors and pro- 
duced an appropriate output. Neural 
Nets are good at untangling complex 
interrelated inputs like this — you 
don’t have to understand the theory 
of the system, just have examples to 
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Figure 5. Image scanning. 
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train the network with. The ANN will sort 
itself out during the training process to make 
sense of the relationships and transforma- 
tions between inputs and outputs. 

One future application for this type of 
‘intelligent sensing’ system is a human to 
machine interface. If you were to try, for 
example, and make realistic prosthetic limbs 
and interface them with part of the nervous 
system, you’d have to try and make sense of 
the outputs of the many nerves (and perhaps 
other biometric inputs and noise), all firing 
with different frequencies. Some of these 
inputs may be irrelevant and others related 
to each other in subtle ways. Processing 
information from this type of complex and 
non-linear system is obviously something 
which is right up the neural net’s ‘street’. 


Recognising Waveforms 


In the man-machine interface discussed 
above, you'd have to process the waveforms 
generated by nerves. And indeed, the Neural 
Net can recognise patterns in time (like 
these) as well as in space (like images), by 
sampling them and feeding them to the 
inputs; Figure 4 shows the idea. 

Of course this leads to the problem, 
already mentioned in part 2, of the image or, 
in this case, waveform, having to be exactly 
the right size and being placed in the middle 
of the grid. 


Pre-processing 


Sizing and centring an image or data, so that 
it’s suitable for a network, is called pre-pro- 
cessing. There are many different ways of 
achieving it. In the case of a sampled wave- 
form, the network is looking at a ‘window’ of 
time, and as the waveform passes it by, one 
sample at a time, it will eventually lie in the 
centre of the window and the network will 
recognise it. 

Exactly the same thing can be said of an 
image-recognising network. It to can also be 
scanned across the image, as shown in Figure 
5. Again, during this process, the pattern to be 
recognised will eventually end up in the cen- 
tre of the grid. Actually this is not dissimilar to 
what our eyes do when we study a scene, we 
(unconsciously) scan the image, looking for 
patterns to recognise. One important point 
about training a network for this sort of task, 
is that we have to train it to recognise ‘noise’ 
or irrelevant data as well as the wanted pat- 
tern or it will give false positives. 

So much for centring the image; what 
about making it the correct size? Well, again 
this can be done by pre-processing. If we 
start with a large window and scan it across 


29 





the image, we can make the window pro- 
gressively smaller, so that sooner or later the 
pattern we want to recognise is correctly 
sized on the grid, see Figure 6. 

Since the neural net inputs are of a fixed 
size, fitting different sizes of image into it 
means manipulating the original image size 
until it fits onto the grid. This is easily done 
by averaging adjacent pixels together until 
the image is the correct size for the network. 


Modular Networks 


The neural networks in the brain recognise 
objects through features. Take the letter L, for 
example. It has two features, a vertical line 
and a horizontal line. No matter where it is 
placed on a grid, nor how small or larger it is, 
it still has these two features. 

If we could have a network which recog- 
nises verticals and another which recognises 
horizontals, then a third can integrate them 
together to form a whole — have a look at 
Figure 7. 

This is a simple example of a modular net- 
work. Rather than one large and fully con- 
nected network trying to learn everything, 
we break the task down into smaller chunks, 


Net 1: Net 2: 
Vertical 
detector 


Horizontal 
detector 


Net 3: 
Combines outputs 
of l and 2 
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Figure 7. Modular networks. 


each one tackled by a smaller net. Research 
has shown that the brain is organised in this 
way — not as one huge net, but as lots of 
smaller modules acting on their own tasks 
and integrating their results together into a 
coherent whole. 

Research is ongoing at the moment to see 
if it is possible to use Evolutionary Algo- 
rithms to grow groups of such modules and 
form an integrated system. 


Neural Function 
The sharp-eyed reader will have spotted that 
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Image information is reduced to 
correct size for network inputs by 
pixel averaging or similar method. 
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Figure 6. Image sizing. 


the description of the Artificial Neu- 
ron given in part 1 doesn't bear too 
much resemblance to the Biological 
one described. In particular, the bio- 
logical neural net frequency-coded 
its activity, as shown in Figure 3. 
Some scientists think that this is 
important and that, what we term 
‘thought’ and ‘consciousness’ are in 
fact sequences of these pulses rac- 
ing around the nervous system, 
interacting with one another. On a 
more pragmatic note, many control 


A neuron which is highly 
stimulated produces many 
pulses per second 
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systems are easier to work with 
when information is coded like this 
(or pulse-width coded). 

Artificial models have been made 
to operate in this manner and are 
often called ‘spiky’ neurons. Figure 
9 shows two simple models, one of a 
‘spiky’ (frequency modulated) neu- 
ron and the other of a pulse-width 
modulated neuron. The weights and 
other parameters of these are easily 
set using a GA (or if you fancy a bit 
of mathematics, by modifying BP). 


A neuron which is only weakly 
stimulated (receiving little signal 
through its dendrites) produces 
few pulses per second 
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Figure 8. Activity in a biological and artificial neuron. 
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Figure 9. Spiky neurons. 


Connections 
and technology 


One final area of interest relates to 
the size of the biological neural net 
compared with the artificial one. You 
may remember that the brain is built 
from 100 billion neurons (the same 
number of stars as there are in our 
galaxy). This easily makes it the 
most complex structure in the 
known universe. 
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On top of this, each of these neu- 
rons may be connected with 1,000 
others (some neurons have 100,000 
connections). This means that there 
are perhaps 100 trillion connections 
in the brain. 

Modern technology cannot match 
either the density of neurons or their 
connections. The neuron is actually 
about the same size as a transistor, 
but is arranged in a dense three 
dimensional structure. Of course, our 


In part 2, some errors have crept into the text which explains Figure 4. Some of 
the equations were spread over two lines and the resulting hyphen is easily con- 
fused with the minus sign. Also, one of the equations contained a mistake. The 
correct equations are given below. 


|. Calculate errors of output neurons 
Errory = out, (| — out,) (Target, — outa) 
Errorg = outg (I — outg) (Targets — outs) 


2. Change output layer weights 
Wag = Wag + Error, outa 
Wag = Wag + nErrorg outa 

W* 30, = Wee + nErrorg out 
Weg = Weg + nErrorg outg 

Wt cg = Weg + Error, oute 
Weg = Wep + nErrorg outc 





3. Calculate (back-propagate) hidden layer errors 
Errora = out, (| — outa) (ErrorgWag, + ErrorgWag) 
Errorg = outg (1 — outg) (Error, We, + ErrorgWpp) 
Errorc = outç (| — outc) (ErrorgWeg, + ErrorgWcg) 


4. Change hidden layer weights 
Waa = Wia + Errora ing 
Won = Woa + NErrorg ing 
taB = Wap + Errore ing, 
"OR = Wop ar nErrorg ino 
tac = Wac + Errore in, 
= Woc ar yErrorc ing 
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circuits are essentially laid flat — two dimen- 
sional. 

It may seem that both the density of neu- 
rons and their wiring are insurmountable 
problems if we wish to make biologically real- 
istic neural nets. But, it should be remem- 
bered that signals travel through real neurons 
at only a couple of hundred metres a second, 
whereas in the artificial equivalent they can 
travel at sixty percent the speed of light. So, 
what the artificial net loses in connectivity, it 
gains in speed. 

It might be possible to solve the connec- 
tivity problem, even with current technology, 
if each neuron is given an address like a host 
in a communications network and this is used 
for routing signals to and from neurons rather 
than having delegated wiring for each signal. 

Making hardware neural nets is a problem 
as there are no integrated circuit technologies 
readily available for the amateur to experi- 
ment with (large discrete neural nets require 
many components) — particularly ones with 
reconfigurable wiring, which is required for 
Evolutionary ANNs. Some experimenters 
have tried Field Programmable Gate Arrays 
(FPGAs), but these are not ideally suited for 
the task as each neuron takes up a great deal 
of chip ‘real estate’. This type of problem 
exists because the neural net was modelled 
on the brain without regard for electronics — 
and so is inefficient to implement. We also 
need to develop a neuron (in the sense of a 
general purpose processing unit, capable of 
learning) which is electronically feasible and 
efficient. 


Finally 


We hope you've enjoyed this trip around the 
world of neural nets enough to try experi- 
menting with some yourself. Listed below are 
some basic references, which are a good 
starting point for further exploration. 
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Further reading 
Some popular books on Neural Nets for those 
who’d like to delve deeper: 


K. Gurney, An Introduction to Neural Net- 
works, UCL Press, 1997. 

|. Pratt, Artificial Intelligence, MacMillan, 1994. 

P. D. Wasserman, Neural Computing: theory 
and practice, Van Nostrand Reinhold, 1989. 


31 


